Skip to content

fix(dev): honor request max_concurrency - #2730

Merged
Luca Forstner (lforst) merged 1 commit into
mainfrom
fix/devserver-request-max-concurrency
Nov 5, 2026
Merged

Luca Forstner (lforst) merged 1 commit into
mainfrom
fix/devserver-request-max-concurrency

Conversation

@paultancre-bt

@paultancre-bt paultancre-bt commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Follow-up to https://github.com/braintrustdata/braintrust/pull/24075, which made the playground and "+ Experiment" dialog send max_concurrency to remote eval dev servers. This PR makes the dev server honor it.

Companion Python SDK PR: braintrustdata/braintrust-sdk-python#1072

Context

The playground Max concurrency slider and the "+ Experiment" Max concurrency field had no effect on remote eval tasks. braintrustdata/braintrust#24075 (merged) made the UI send max_concurrency in the POST /eval request to the dev server, but evalBodySchema stripped the field and the /eval handler spreads ...evaluator into Eval(), so it was ignored. The value set on the Eval in code always won. A customer lowered the slider to avoid Azure 429s and saw no change (Pylon 20534, COR-288).

Description

Parse max_concurrency from the eval request and pass it to Eval() as maxConcurrency.

The request value can lower the evaluator's limit but never raise it (min(request, evaluator); if the evaluator sets no limit, the request value applies). This is deliberate: the playground always sends a value and the slider defaults to 10, so letting the request override outright would silently raise every code-set limit below 10 (e.g. max_concurrency=5 → 10) as soon as users upgrade the SDK. The trade-off is that the UI can't push concurrency above what's in code.

Other details:

  • null or a missing value keeps the evaluator's limit, which also covers older UIs that don't send the field.
  • Fractional values are rounded down (the "+ Experiment" field is a free number input validated only as >= 1). Values below 1 or non-numbers are rejected with a 400.

Testing

  • New tests cover: request lowers the limit, request can't raise it, request applies when the evaluator has no limit, null or omitted request keeps the evaluator's limit, rounding down, and invalid values. They fail without the fix and pass with it.
  • vitest run dev/server.test.ts (13 passed), pnpm run check:typings, and prettier are clean. Changeset included (patch).
  • End to end: ran the real dev server (eval --dev) from this branch against a Braintrust org, with a 12-row eval whose task records how many calls are in flight, and sent POST /eval requests shaped like the playground's. Same run against the released JS 3.42.0 (npm) for comparison:
Evaluator limit in code Request max_concurrency Expected peak This branch Released
5 2 2 2 5
5 10 5 5 5
5 null 5 5 5
5 omitted 5 5 5
5 2.5 2 2 5
none 3 3 3 12
none omitted 12 12 12

Refs: COR-288

The playground and "+ Experiment" dialog now send max_concurrency to
remote eval dev servers (braintrust#24075), but evalBodySchema dropped
it, so lowering the slider had no effect on remote evals.

Parse max_concurrency from the eval request and pass it to Eval(),
capped at the evaluator's own maxConcurrency. The request can lower
the limit but never raise it, so the playground's default of 10
cannot exceed a limit set in code.

Refs: COR-288

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@lforst
Luca Forstner (lforst) merged commit 245d1f9 into main Nov 5, 2026
63 checks passed
@lforst
Luca Forstner (lforst) deleted the fix/devserver-request-max-concurrency branch November 5, 2026 12:11
Abhijeet Prasad (AbhiPrasad) pushed a commit to braintrustdata/braintrust-sdk-python that referenced this pull request Nov 5, 2026
Follow-up to braintrustdata/braintrust#24075,
which made the playground and "+ Experiment" dialog send
`max_concurrency` to remote eval dev servers. This PR makes the dev
server honor it.

Companion JS SDK PR:
braintrustdata/braintrust-sdk-javascript#2730

### Context

The playground **Max concurrency** slider and the "+ Experiment" **Max
concurrency** field had no effect on remote eval tasks.
braintrustdata/braintrust#24075 (merged) made the UI send
`max_concurrency` in the `POST /eval` request to the dev server, but
`parse_eval_body` never read the field and `run_eval` built the
`EvalAsync` kwargs only from the registered evaluator, so it was
ignored. The value set on the Eval in code always won. A customer
lowered the slider to avoid Azure 429s and saw no change (Pylon 20534,
COR-288).

### Description

Parse `max_concurrency` from the eval request and pass it to
`EvalAsync`.

**The request value can lower the evaluator's limit but never raise it**
(`min(request, evaluator)`; if the evaluator sets no limit, the request
value applies). This is deliberate: the playground always sends a value
and the slider defaults to 10, so letting the request override outright
would silently raise every code-set limit below 10 (e.g.
`max_concurrency=5` → 10) as soon as users upgrade the SDK. The
trade-off is that the UI can't push concurrency *above* what's in code.

Other details:
- `null` or a missing value keeps the evaluator's limit, which also
covers older UIs that don't send the field.
- Fractional values are rounded down (the "+ Experiment" field is a free
number input validated only as `>= 1`). Values below 1 or non-numbers
are rejected with a 400.

### Testing

- New tests cover: request lowers the limit, request can't raise it,
request applies when the evaluator has no limit, null or omitted request
keeps the evaluator's limit, rounding down, and invalid values. They
fail without the fix and pass with it.
- `nox -s test_cli` passes (43 passed, 2 skipped); `make lint` is clean.
- End to end: ran the real dev server (`eval --dev`) from this branch
against a Braintrust org, with a 12-row eval whose task records how many
calls are in flight, and sent `POST /eval` requests shaped like the
playground's. Same run against the released Python 0.49.2 (PyPI) for
comparison:

| Evaluator limit in code | Request `max_concurrency` | Expected peak |
This branch | Released |
|---|---|---|---|---|
| 5 | 2 | 2 | **2** | 5 |
| 5 | 10 | 5 | 5 | 5 |
| 5 | `null` | 5 | 5 | 5 |
| 5 | omitted | 5 | 5 | 5 |
| 5 | 2.5 | 2 | **2** | 5 |
| none | 3 | 3 | **3** | 12 |
| none | omitted | 12 | 12 | 12 |

Refs: COR-288

Co-authored-by: Paul Tancre <paultancre@p-tancre.home>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants